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A complete mathematical theory of nonequilibrium thermodynamics of stochastic systems in 
terms of master equations is presented. As generalizations of isothermal entropy and free energy, 
two functions of state play central roles: the Gibbs entropy S and the relative entropy F, which are 
related via the stationary distribution of the stochastic dynamics. S satisfies the fundamental en- 
tropy balance equation dS/dt — e v — hd/T with entropy production rate e p > and heat dissipation 
rate hd, while dF/dt — —fd < 0. For closed systems that satisfy detailed balance: Te p (t) = fd(t). 
For open systems one has Te v (t) — fd(t) + Qhk(t) where the housekeeping heat, Qhk > 0, was 
first introduced in the phenomenological nonequilibrium steady state thermodynamics. Entropy 
production e v consists of free energy dissipation associated with spontaneous relaxation (i.e., self- 
organization), fd, and active energy pumping that sustains the open system Qhk- The amount of 
excess heat involved in the relaxation Q ex = hd ~ Qhk = fd ~ T(dS/dt). 



PACS numbers: 

Thermodynamics is the mathematical theory that de- 
scribes the transformations of energy among all its forms. 
In particular, entropy and free energy are key quantities 
in understanding energy transformations at finite tem- 
perature. J.W. Gibbs' statistical mechanics, which estab- 
lished the relation between molecular systems and their 
equilibrium thermodynamics, is founded on the assump- 
tion that isothermal molecular systems are stochastic, 
and macroscopic quantities such as energy and number 
of particles can, and should be treated as random vari- 
ables with distributions in terms of the concept of ensem- 
ble. In Gibbs' approach, the fundamental origin of the 
stochasticity, in the surrounding "bath" , is left for others 
to ponder, while he spearheaded into applications of his 
approach to complex physical systems such as chemical 
solutions, systems too complex to be rigorously studied 
by the kinetic theory of Boltzmann |l| . 

The Gibbs' approach can be naturally generalized in 
two different directions: The relaxation dynamics of a 
system in contact with an equilibrium bath (iso-thermal, 
iso-chemical potential, etc.) and the steady state of a 
system in contact with a nonequilibrium environment 
with sustained chemical potential difference (i.e., an open 
chemical systems [U, Q). In recent years, it becomes in- 
creasingly clear that both theories can be framed in terms 
of the mathematical theory of Markov processes [3, @ . 
The significant progresses in both fluctuation theorems 
in terms of Markov models and 1-dimensional exclusion 
processes in terms of interacting particle systems testify 
the centrality of the probability theory 0, Q . 

A thermodynamics theory has emerged from the math- 



ematical analysis of Markov processes @. The present 
paper gives the first complete account of this emergent 
mathematical structure. In addition, we also raise sev- 
eral issues concerning the nature of dissipation, thus, ir- 
reversibility, and its connection to measurement of heat. 

To avoid intricate mathematical techniques, we con- 
sider Markov systems with discrete state variables and 
continuous time, characterized by master equations. The 
1976 review [5] provides a natural starting point. Gener- 
alization of the our present results to continuous variables 
in terms of Fokker-Planck equations is straightforward. 



MASTER EQUATIONS WITH DETAILED 
BALANCE 



We shall consider a molecular system in terms of a 
Markov model, dpi(t)/dt = Y^,j (PjQji ~ PiQij), that is ir- 
reducible: Hence there is a unique long-time stationary 
probability distribution {pf}. One class of master equa- 
tions is particularly important: Its stationary distribu- 
tion satisfies detailed balance: pfqij = PjQji, where q^ is 
the transition probability rate from state i to j. 

The following results are well known: 

(i) The system has an internal energy m = — Thxpf 

[lfj, and one can define an entropy S, a total internal 
energy U, and a free energy F, all functions of the state 
of the system 

S[{pi}] = -^P<lnpi, U[{pi}] = ^2piUi, 

i i 

F = u-TS = Tj2m^(^). (1) 



'Electronic address: 
t Electronic address: 



gchao@fudan.edu.cn 



qian@amath.washington.edu 



Throughout the present work, ks = 1 and the tempera- 
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(ii) For Pi(t) as the time-dependent solution to the 
master equation, all U, S and F are functions of t, and 

dU[{ Pi (t)}} _ , 



dt 



PjQji 



(2) 



— 6r> 



(Pi3« ~ PjQji) In 



d5[{Pift)}] 
dt 



Te p (t) - h d (t). 



(3) 
(4) 



©-© have very clear thermodynamic meanings: The 
total internal energy of the system changes due to ex- 
change heat with the bath and the free energy of 
an isothermal system spontaneously decreases until it 
reaches to its minimum at equilibrium. It can be shown 
that > 0. The term e p is called entropy pro- 

duction rate; it is the same are free energy dissipation 
rate. The entropy of the system increases due to entropy 
generated in spontaneous processes and decreases when 
heat is expelled into the surrounding. Note in isothermal 
system, heat and entropy are related by a simple factor 
(temperature) due to the Clausius equality. 

Therefore, there is a complete satisfying time- 
dependent thermodynamics for system in contact with 
an equilibrium bath relaxing to equilibrium. "In contact 
with an equilibrium bath" is mathematically represented 
by the Wegscheider-Kolmogorov cycle conditions on the 
Qij's BE- 



II. TWO GENERALIZATIONS OF THE 
THERMODYNAMICS TO SYSTEMS WITHOUT 
DETAILED BALANCE 

Many of the above results can be generalized to mas- 
ter equations without detailed balance. The only relation 
that no longer exists is detailed balance: pf/pj ^ Qij/Qij 
where we denote the unique steady state probability dis- 
tribution as {pf}. Again, we can still phenomenolog- 
ically introduce Ui = — Tlnpf 13], total energy U — 
J2iPi u ii Gibbs entropy S = Y^iPi^Pii an d free energy 
F = U-TS = TJ2iPi In (Pi/Pi)- The last term is widely 
known as relative entropy. 

Thermodynamics based on Gibbs' entropy. The 
Gibbs' entropy is a generalization of Boltzmann's for- 
mula to situations with nonuniform probability distribu- 
tions. According to the the fundamental postulate of 
nonequilibrium thermodynamics of de Groot and Mazur 
[l2| , the entropy change dS can be distinguished between 
two terms: d e S is the transfer of entropy across the 
boundaries of the system, and diS is the entropy pro- 
duced within the system due to spontaneous processes. 
In terms of the g^ , we thus have for isothermal systems 

dS({pi(t)}) d l S d e S h d (t) 

dt " + " epW ~^' (5) 



where e p (t) = 3 V,., :/'.'/-., PjQji) ^ (jHr) is the 
instantaneous entropy production rate, and h d (t) = 

i" J2i j (PiQij ~ PjQji) m f f^") ^ s ^ ne heat dissipation rate. 
The e p is the focus of recent intense studies on fluctuation 
theorem 

Thermodynamics based on relative entropy. It is 

widely known in the mathematical literature that F > 
and dF({pi(t)})/dt < 0, both are hold for master equa- 
tion without detailed balance 0]. In fact, in terms of the 
qij , the free energy dissipation fa 



dF({ Pl (t)}) 
dt 



= -fd 



(6) 



Many people have noticed this nice Lyapunov properties: 
14| used it in their ii-theorem for stochastic dynamics, 
5| discussed it in connection to Prigogine-Glansdorff 's 
criterion of steady-state thermodynamic stability. Both 
Reguera, Rubi and Vilar, and Ao have constructed their 
thermodynamic theories, the mesoscopic nonequilibrium 
thermodynamics and the Darwinian dynamics respec- 
tively, based on Eq. [6] Mackey has presented the dy- 
namic origin of the arrow of time based on F ||. 

Relationship between e p and fd and decomposi- 
tion of heat dissipation. Without the detailed bal- 
ance, fd and e p are no longer identical, even though both 
have many important properties. Both are nonnegative; 
both follow their own fluctuation theorems [l5|, Ha ] 

We discover that the difference between e p and f d is in 
fact the "housekeeping heat" , 



T 

Qhk = Te p -fd= — ^ [piQij - PjQji) In 



^,3 



PiQij 
rfQji 



, (7) 



first put forward by Oono and Paniconi in a purely 
phenomenological theory of nonequilibrium steady state 
(NESS) thermodynamics 17). They decomposed the to- 
tal heat dissipation into a "housekeeping" part and an 
"excess" part. In their theory, the irreversibility of a 
process converting work into excess heat when modulo 
housekeeping heat. Later, Hatano and Sasa combined 
this concept with Langevin dynamics and established 
a deep connection between the phenomenological NESS 
thermodynamics and the Jarzynski equality jl7j. One 
of us recently has generalized their extended form of the 
Second Law to abstract Markov processes [18| • 

By convention, we shall take the sign of heat to be 
positive when it flows from the system to its environment. 
Then, our hd is the total heat dissipation, and the excess 
heat 

Qex = hd — Qhk — h d —Te p +fd — fd—T — ^ , (8) 

dt 

which represents the "hidden" heat term that is involved 
in the "driving mechanism" of the open system. Accord- 
ing to this phenomenological view, a NESS by definition 
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is not driven. It is important to point out that the entire 
NESS thermodynamics based on and Q ex is nonlo- 
cal: This can be seen from the Eq. [7] which contains the 
infinitely long time {pf} in its definition. For stochastic 
system without detailed balance, {pf} in principle can 
not be constructed locally as done following the Boltz- 
mann's Law. 
We also have 



inequalities: 



T 



— fd + Qhk — — 



dF[{Pi(*)}] 
dt 



Qhk 



(9) 



This implies that there are two different origins for the to- 
tal entropy production: — dF Jf* is from the spontaneous, 
non-stationarity and Qhk(t) from the driven mechanism 
that sustains the nonequilibrium environment. They are 
both traditionally called "nonequilibrium" and were not 
distinguished before. The decomposition of the two pro- 
vides a deeper understanding of irreversibility, as we shall 
show next. 



III. DIFFERENT MATHEMATICAL 
STATEMENTS OF THE SECOND LAW BASED 
ON NONNEGATIVE e p AND f d 

The Second Law of Thermodynamics for isothermal 
system is about decreasing free energy, not increasing en- 
tropy. For open systems, we still have the two important 
quantities satisfying 

T d£g) = Tep (fi - hd ( t ) = f d ( t ) - Q„(ty, (10a) 



dt 
dF(t) 
dt 



= Qhk{t)-Te p (t) = -f d (t). 



(10b) 



This much is mathematically concrete. Heuristically, we 
note that hd = —TdS e /dt is the heat dissipated via the 
boundaries to its environment. Hence, an experimen- 
tal observer who resides "inside" the system might not 
be able to aware of the dissipative nature of the "driv- 
ing mechanism". In that case, he/she might be able 
to construct a "conservative dynamics" and a complete 
analogue to equilibrium thermodynamics based on fd- 
The fd explicitly characterizes the spontaneous relax- 
ation processes. This is indeed the thesis of Oono and 
Paniconi, Ao, and Wang [l,[l||. However, to an observer 
outside the system, as most cell biologists studying a cell, 
the relaxation is a form of self-organization driven by the 
external energy. 

Even in the steady state, there are still nonzero e p and 
hd' There is a continuous useful energy being pumped 
into the system, that sustains the system in a NESS. 
We think the issue here is rather deep and philosophical; 
hence we shall defer further discussions elsewhere. 

From the definition of e p given above, it is clear that 
s-p(t) > and the equality holds if and only if the system 
has detailed balance. Then, immediately we have several 



dS(t) , h d (t) 
dt 
dF{t) 



T 



e„(t) > 0, 



dt 



Qhk(t) = -T-e p (t) <0. 



(11a) 



(lib) 



Eq. Illal is just the well-known Clausius inequality 
(dS > —Qtot/T), which could be rectified through the 
quasi-stationary process to obtain expressions for the 
entropy produced (dS) as the result of heat exchanges 
(Qtot = hd). And Eq. Illbl is a general version of the free 
energy inequality for the amount of work performed on 
the system, since the work values must then be consis- 
tent with the Kelvin-Planck statement [2(| and forbids 
the systematic conversion of 100% heat to work isother- 
mally. 

More precise, the quantity Qhk(t) in Eq. Illbl van- 
ishes when the detailed balance condition, pf{t)qij(t) — 
Pj(t)qji(t), holds and then it returns to the traditional 
Helmhotz or Gibbs free energy criterion of equilibrium 
thermodynamics, depending on whether a it is a NVT 
or NPT system [2l|. 

According to the nonnegativity of fd, one has 



T-^M+Q ex (t) = f d (t)>0, 



dt 



dF(t) 
dt 



-T-e p (t)+Q hk (t)<0. 



(12a) 
(12b) 



Eq. I12al is the extended form of Clausius inequality dur- 
ing any nonequilibrium time-dependent process, which 
is a specail case of Hatano and Sasa's latest work [17]. 
And Eq. I12bl provides an alternative form of free energy 
inequality. 

The housekeeping heat is also nonnegative, which im- 
plies the driven nature of the system and it is in fact a 
measure of how far the system is kept away from detailed 
balance. This measure is defined not only to NESS, but 
also for spontaneous relaxation to NESS. 



IV. DISCUSSION 

Equilibrium statistical mechanics of Gibbs is one of 
the corner stones of theoretical physics. Classical ther- 
modynamics, however, is applicable to both equilibrium 
and nonequilibrium systems. In fact the most cele- 
brated statement of The Second Law of Thermodynamics 
concerns with an irreversible, macroscopic, spontaneous 
molecular process. 

There are two types of nonequilibrium processes which 
have been widely studied: those of time-dependent na- 
ture and those with stationary characteristics. The theo- 
ries of transport and of Zwanzig-Kubo belong to the first 
type. The irreversibility of this type of processes is orig- 
inated in their initial conditions. They are of transient 
nature. The second type requires a sustained, continuous 
energy supply; it is driven. One of the most important 
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examples is a living cell in its homeostasis [2l| . This type 
of systems have been called nonequilibrium steady state 
(NESS) [6]. Oono and Paniconi have developed a rather 
rigorous phenomenological NESS thermodynamics [i~7l ]. 
One of the most important concepts from their analysis 
is the decomposition of total dissipated heat into Qhk 
and Qex- 

Following the tradition of Gibbs, an investigation of 
thermodynamics of molecular systems has to be based 
on the mathematical theory of stochastic processes. The 
recently developed stochastic thermodynamics has ex- 
tended the field much further than other approaches 



[22I [23|]. There are fundamental differences in the ir- 
reversible thermodynamics of driven systems and irre- 
versibility of a spontaneous process. An important con- 
cept, the housekeeping heat, has emerged in recent studies 
of such systems and processes. For non-driven isother- 
mal systems, also widely known as closed system, the 
free energy dissipation, which characterizes spontaneous 
irreversibility, and the entropy production, which char- 
acterizes the total irreversibility, are the same. However, 
for driven (open) system, there is a difference between 
these two quantities: The housekeeping heat is a mea- 
sure of the difference. 
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